Skip to content

perf: optimize to_char in datafusion-functions#23544

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/to_char-datafusion-20260713-194530
Draft

perf: optimize to_char in datafusion-functions#23544
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/to_char-datafusion-20260713-194530

Conversation

@andygrove

@andygrove andygrove commented Jul 14, 2026

Copy link
Copy Markdown
Member

This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.

Which issue does this PR close?

N/A — autonomous exploratory PR.

Rationale for this change

Cache ArrayFormatter per distinct format string (and per retry format) in to_char's array-format path instead of rebuilding one for every row, so each format is parsed once rather than N times.

What changes are included in this PR?

Cache ArrayFormatter per distinct format string (and per retry format) in to_char's array-format path instead of rebuilding one for every row, so each format is parsed once rather than N times.

Are these changes tested?

Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).

Benchmark (criterion):

  • to_char_scalar_datetime_pattern_1000: -0.298% faster (base 145108ns -> cand 145541ns)
  • to_char_array_date32_datetime_patterns_1000: 8.589% faster (base 316946ns -> cand 289724ns)
  • to_char_scalar_date_only_pattern_1000: -0.91% faster (base 72981ns -> cand 73645ns)
  • to_char_array_date32_mixed_patterns_1000: 7.606% faster (base 214980ns -> cand 198629ns)
  • to_char_array_date_only_patterns_1000: 3.603% faster (base 106325ns -> cand 102494ns)
  • to_char_array_datetime_patterns_1000: 8.85% faster (base 185444ns -> cand 169032ns)
  • to_char_array_mixed_patterns_1000: 11.564% faster (base 153095ns -> cand 135392ns)

Full criterion output:

to_char_array_date_only_patterns_1000
                        time:   [102.42 µs 102.62 µs 102.82 µs]
                        change: [−3.9223% −3.6032% −3.3220%] (p = 0.00 < 0.05)
                        Performance has improved.

to_char_array_datetime_patterns_1000
                        time:   [168.71 µs 169.00 µs 169.28 µs]
                        change: [−9.0254% −8.8501% −8.6763%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

to_char_array_mixed_patterns_1000
                        time:   [134.99 µs 135.35 µs 135.72 µs]
                        change: [−11.792% −11.564% −11.331%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

to_char_scalar_date_only_pattern_1000
                        time:   [70.998 µs 74.479 µs 77.713 µs]
                        change: [−4.1934% +0.9103% +6.4073%] (p = 0.74 > 0.05)
                        No change in performance detected.

to_char_scalar_datetime_pattern_1000
                        time:   [139.28 µs 145.54 µs 151.23 µs]
                        change: [−4.8324% +0.2984% +6.0502%] (p = 0.90 > 0.05)
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  14 (14.00%) low mild

to_char_array_date32_datetime_patterns_1000
                        time:   [289.36 µs 289.80 µs 290.33 µs]
                        change: [−8.7478% −8.5888% −8.4279%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

to_char_array_date32_mixed_patterns_1000
                        time:   [197.15 µs 197.85 µs 198.56 µs]
                        change: [−8.0355% −7.6059% −7.1932%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Are there any user-facing changes?

@github-actions github-actions Bot added the functions Changes to functions implementation label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

functions Changes to functions implementation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andygrove